Simple Recurrent Networks and Natural Language: How Important is Starting Small?

Prediction is believed to be an important component of cognition, particularly in the processing of natural language. It has long been accepted that recurrent neural networks are best able to learn prediction tasks when trained on simple examples before incrementally proceeding to more complex sentences. Furthermore, the counter-intuitive suggestion has been made that networks and, by implication, humans may be aided during learning by limited cognitive resources (Elman, 1991 Cognition). The current work reports evidence that starting with simplified inputs is not necessary in training recurrent networks to learn pseudo-natural languages. In addition, delayed introduction of complex examples is often an impediment to learning. We suggest that special teaching methods and limited cognitive resources during development may be of no benefit for learning the structure of natural language.